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Some  of  the  analyses  performed  in  this  report  were  done  on  the  PLAT0R 
system.  The  PLAT0R  system  is  a  development  of  the  University  of  Illinois 
and  PLAT0R  is  a  service  mark  of  Control  Data  Corporation. 


Tatsuoka  (1983)  showed  in  the  context  of  tigned-nueber  probleet  that 
dividing  the  unit  of  tcorei  into  subcomponents,  tign  and  nueber  parte, 
enabled  ue  to  observe  Misconceptions  possessed  by  students  explicity. 

However,  the  recent  development  of  computer  technology  has  made  it  easier  to 
write  computer  programs  that  are  capable  of  solving  problems  in  procedural 
domains.  This  study  utilizes  such  work  to  show  cognitive  errors  occurring 
at  unobservable  subcomponents  of  problem-solving  activities. 

Riley  and  her  associates  (Riley,  1983;  Riley,  Bee  V  Hokwa,  1981)  have 
developmd  a  model  that  represents  the  theoretical  analysis  of  the  knowledge 
required  for  successful  performance  on  the  Navy's  basic  electricity  and 
electronics  (BEtiE)  training  program.  The  analysis  is  presented  in  the  fora 
of  a  computer  program,  in  other  words,  a  production  system  which  is  comprised 
of  two  components  of  knowledge.  The  first  component  is  a  database  that 
includes  a  network  of  elements  and  relations  between  those  elements.  The 
second  is  a  set  of  production  rules  which  consist  of  conditions  and 
actions  related  with  "if"  and  "then*  statements.  A  schematic 
representation  of  solving  problems  in  D-C  circuits  is  presented  in  one  of 
Riley's  studies  (Riley,  1983).  The  processes,  components  of  the  problem 
and  their  relations  are  expressed  in  a  coherent  structure  in  her  paper. 
Further,  an  empirical  study  through  a  series  of  interviews  with  students  at 
the  Navy  BEfcE  school  revealed  that  the  lack  of  understanding  and 
acquisition  of  important  components  of  knowledge  structure  yielded  several 
errors  which  were  also  identified  by  the  theoretical  analysis.  Montague 
and  Riley  constructed  a  new  test  by  taking  these  findings  into  account. 

This  study  uses  the  Montague-Ri ley  test  to  introduce  a  new  scoring 
procedure  that  enables  us  to  see  errors  in  cognitive  processes  occurring  at 
subcomponents  of  solving  an  electricity  problem.  The  scores  from  the  new 


2 


■coring  procedurt  reflect  the  weaknesses  and  strengths  of  a  student's 
understanding  in  the  important  knowledge  components. 

Hontague-Ri ley  test  and  a  computer  program  that  solves  a  circuit  problem 
The  tests  consist  of  four  parts.  Each  part  includes  36  problems. 

The  content  of  the  test  is  divided  intai  1)  series  circuits! 

2)  parallel  circuits;  3)  and  4)  combination  circuits.  In  this  study,  parts 
1)  and  2)  (simple  circuits)  are  subjected  to  investigation.  As  can  be 
seen  in  Figure  1,  the  problems  are  open-ended  and  require  several 


Insert  Figure  1  about  here 


steps  to  reach  the  right  answers.  These  steps  aret  1)  knowing  the 
terminology  and  its  meanings)  2)  understanding  the  schematic  expression 
of  a  circuit!  3)  setting  the  known  conditions  properly  that  are  given  in 
the  problem;  4>  select  an  unknown  variable  that  can  be  solved  from  the 
known  condition;  3)  select  appropriate  rule(s)  expressing  various 
relations  among  resistors,  current  and  voltage;  6)  substitute  proper 
numerals  or  relations  (increase  or  decrease)  in  the  variables  ;  7)  solve 
the  equation  for  the  unknown  variable;  or  make  a  quantitative  inference; 

8)  get  the  answer;  9)  update  a  set  of  known  conditions. 

Independently  from  Riley's  production  system,  a  new  computer 
program  was  designed  and  written  on  the  PLATO®  system  at  the  University 
of  Illinois  (ELTEST,  Tatsuoka  fc  Baillie,  1984).  Riley's  production 
system  uses  a  backward-search  strategy  to  locate  the  requested  unknown 
variables  which  are  solvable  by  the  known  conditions,  but  the  ELTEST  program 
applies  a  decision-table  strategy  which  consists  of  a  contingency  table 
of  four  rows  and  n  columns  (the  number  of  resistors  plus  two  columns).  The  first 


row  represents  resistors  Rj ,  the  second  tor  voltage  drop  Ej  at  R^  and 
the  third  row  for  the  current  at  Rj ,  i>l,2,...,n.  The  fourth  row 
contains  the  number  of  total  cells  filled  in  the  known  quantities  or  the 
known  relations!  increase,  decrease  or  unchanged.  The  n-th  coluen 
represents  the  total  resistance,  total  voltage  drop,  and  the  total 
current.  The  last  coluen  includes  the  number  of  cells  filled  in  the 
known  quantities  or  relations.  The  information  in  these  last  rows  and 
columns  enables  the  computer  program  to  solve  a  problem  forwardly 
instead  of  backwardly  as  in  Riley's  production  system.  The 
forwarding  path,  the  ordered  string  of  consecutively  solvable  variables 
is  uniquely  determined  except  for  that  Ej  and  Ej  tidj)  may  be  alternated. 
Riley's  theoretical  findings  that  the  lack  of  understanding  in  the 
part-whole  relations  properly  causes  errors  in  the  answers  was  taken 
into  acount  in  the  design  of  ELTEST.  The  part-whole  relationships  are 
decomposed  into  two  parts;  that  of  between  columns  and  within  columns 
(Ohm's  law). 

The  between-column  relations  include! 


n  n 

E  Rj  ■  Rt,  I  Ej  «  Efc,  Ij  *  12  *  *  It  *or  circuit, 

i*l  i«l 


n  n 

E  1/R;  «  1/Rt,  Ej  ■  E2  ■  •••  "  Ef  E  Ij  «  It  for  parallel  circuits 
i  =  1  i»l 


and  the  withln-column  relations  common  to  both  the  series  and  parallel 
circuits  (Ohm's  law)  are  given  by, 


I »  *  E • / R ■ ,  i*l ,2,. . . ,n. 


Hi aconcaptiona  or  lack  of  knowladqa 

Marty  arrora  can  ba  anticipatad  at  any  atep  froa  1  to  8  daacribad 
aarliar  by  a  contant  analyaia.  Howavar,  Rilay  (1984)  found  fiva  aourcaa 
of  aiaconcaptiona  through  har  protocol  atudy.  Sha  raportad  tha 
fraquanciaa  of  thaaa  arrora  with  a  aaapla  of  7  Naval  trainaaa.  Thasa 

arrora  ara  daacribad  in  Tabla  1  with  tha  aaaa  coda  aha  uaad  in  tha 

* 

pravioua  atudy. 


Inaart  Tabla  i  about  hara 


Tha  Hontagua-Rilay  taat  waa  adainiatarad  to  230  high  achool  atudanta. 
Fraquanciaa  of  thair  raaponaaa  to  tha  taat  itaaa  and  fraquanciaa  of  thair 
raaponaa  pattarna  aa  a  whola  war  a  calculatad,  and  tha  high-f raquancy 
raaponaaa  ara  auaaarizad  in  Tabla  2. 


Inaart  Tabla  2  about  hara 


Aa  can  ba  aaan  in  Tabla  2,  quita  a  faw  atudanta  producad  arronaoua 
raaponaa  pattarna  diffarant  froa  thoaa  of  tha  right  raaponaaa.  Tha 
purpoaa  of  tha  naxt  aaction  ia  to  idantify  tha  undarlying  aiaconcaptiona 
which  raaultad  froa  producing  thoaa  anawara.  Firat,  a  naw  acoring 
procadura  (coaponant  acoring  procadura)  that  ia  daaignad  to  raflact  cloaaly 
tha  raaulta  of  thaoratical  analyaia  aantionad  aarliar  into  tha  acoraa  of 
nina  raaponaaa  will  ba  introducad. 

Coaponant  acoring  procadura 


Tatauoka  (1983),  Tatauoka  and  Tatauoka  (1981)  propoaad  a  coaponant 
acoring  procadura  in  ai gnad-nuabar  aubtraction  problaaa  (nuafcar  and  aign 


part*  of  raponsaa  are  acorad  separately) »  and  foreulated  a  rula  apaca  where 
each  arroneoua  rula  of  operation  waa  preaented  aa  an  ordered  pair  of  latent 
variablea  8j  and  6 2 »  representing  the  abilities  of  calculating  the  correct 


Table  1 

Riley's  Five  Categories  of  Errora 

1.  G C  —  1  Failure  to  dietinguish  the  identitiea  of  the  quantitiea 


Mapped  into  circuit  foreulas. 


2. 

E  *  R 

Failure  to  dietinguish  between  voltage  drops 

and  resistance. 

3. 

LG 

Mixing  local  and  global  variables  when  OhM's 

law  was  solved. 

4. 

NA 

No  Answer 

5. 

Other 

Table  2 


Frequencies  of  rt«pon»t  pattern!  observed  in  the  probleat  of  »erit» 


circuits  (N  ■  2S3) 


Problee  i 

Frequencies 

Response 

Pattern 

132* 

+-+-0 - 

5 

♦0+0+---- 

20 

♦0+0+0000 

S 

♦“+“00000 

6 

+0000—  — 

5 

♦-+-+0000 

6 

+-0-0 - 

3 

♦ 

1 

1 

1 

o 

1 

1 

1 

1 

Problee  3 


Problee  2 

Frequencies 

Response 

Pattern 

108* 

“+—+0++++ 

3 

-♦-♦00000 

14 

-0-0-0000 

34 

-+-+0+-++ 

Problee  4 


Frequencies 


Response 


Frequencies 


Response 


numbers  and  signs,  respect! vely ,  for  the  answers.  However,  the  problems  in 
the  Montague-Ri ley  teat  ara  far  aora  complicated  than  signed-number 
arithmetic  and  thua  require  the  help  of  a  computer.  For  example,  Problem  1 
in  the  aeriaa  circuit  taat  make: 

What  mould  ba  the  affect  of  an  increaae  in  reaiatance  of  R2  on 

the  circuit  values? 

The  atudenta  are  expected  to  fill  in  the  boxes  diaplayed  in  the  problem 
with  "+"  for  increaae,  for  decreaae,  and  "o"  for  remain  fixed.  The 
right  procedure  starts  with  filling  cell  R2  with  *+“  and  Ea  with  "0"  because 
the  battery  voltage  (Ea)  should  always  ba  invariant.  Then  the  cells  R1 
and  R2  must  be  filled  with  "0".  Figure  2  shows  a  summary  of  the  successive 
conditions  determined  by  various  stages  of  problem  solving.  The  only 
solvable  equation  in  No.  1  of  Figure  2  is  the  relation  of  the  first  row: 

Rj  +  R2  +  R3  =  C  1  and  the  answer  is  "+“.  Then  the  next  step  is  to 


Insert  Figure  2  about  here 


apply  Ohm's  law  at  the  total  level  in  No.  2:  I ^  =■  Ea/Rt  and  the  right 
answer  is  ,,-u.  The  next  step  also  follows  uniquely:  use  the  relation  Ij  = 
Ij  *  I3  «  1^  (because  the  current  is  invariant)  to  fill  the  cells  Ij,  I2 
and  I3.  Now  in  No.  3  of  Figure  1,  apply  Ohm's  law,  Ej  ■  Ij  x  Rj  and  E3  * 

I3  x  R3.  Then  the  answers  in  cells  Ej  and  E3  are  uniquely  determined  and 
they  are  Since  Ohm's  law  at  E2,  E2  *  I2  x  R2  cannot  provide  the 

unique  answer  with  the  conditions  R2  »  and  I2  ■  the  relationship  Ey  ■ 


Ej  ♦  E2  +  E3  nust  be  used.  Since  Ey  is  invariant,  Ej  and  E3  are 
decreasing,  the  part-Nhole  relation  detereines  E2  to  be  +  .  Thus,  the 
response  sequence  to  the  problem  is  "♦  -  +  -  0  -  -  -  when  the  right 
procedure  is  applied.  The  procedure  used  in  solving  each  unknown  is 
summarized  in  the  Table  3. 


Insert  Table  3  about  here 


The  computer  program  ELTEST,  described  in  an  earlier  section,  was  used 
to  score  230  students'  responses  to  the  items.  The  program  generates  the 
right  answer  consecutively  for  an  unknown  variable  encountered  and  solves 
it  at  each  step  by  using  the  conditions  determined  by  the  previous  steps. 

If  the  generated  response  to  the  question  does  not  match  a  student's 
answer,  then  the  program  prints  the  location  of  the  component  as  well  as 
the  conditions  used  at  the  current  step,  picks  up  the  student's  wrong 
response  and  makes  it  one  of  the  conditions  in  solving  the  next  unknown 
variable  (question).  The  sequential  path  of  solving  each  unknown  variable 
can  be  examined  by  using  Table  4.  A  row  of  the  table  represents  the 


Insert  Table  4  about  here 


necessary  conditions  to  be  known  in  order  to  obtain  the  value  of  a  column 
variable  with  one  type  of  four  arithmetic  operations  without  any 
combination.  For  example,  E j  ■  E 4  ■  -E2  -  E3  is  a  simple  operation  but 
E2  »  (E+/R*)-R2  is  a  combination.  The  variation  of  paths  of  solving  unknown 
variables  are  only  within  a  step,  hence  in  terms  of  scoring  applications  of 
rules  the  path  difference  is  trivial  and  does  not  affect  component  scoring 


Tabic  3 


Suaaary  of  the  right  procedure  at  each  ctcp  of  solving  the  unknown! 
Variable  Current  Relation  Answer  Order  of 

Condition  to  be  used  etepe 


*t 

Rj»0,  R2m+*  R3m<MEtl0) 

Rt-R i+R2+R3 

:1 

VO 

M 

O 

II 

1 

e 1  *  1 1  *  Ri 

^2  ( 

^  R 2 *+ i l2"~ 

e2"^2  x  r2 

El  —  «  e3"“»  Et“° 

Et-E1+E2+E3 

lZ 

R3*0 1  13s- 

E3**3  *  *3 

lt 

Et«0 

Tereinology 

h 

O 

II 

It-Il 

[2 

rt* 

II 

O 

*-4 

H 

O 

'3 

It"0* I 1"0* I2“0 

It*,1"I2"I3 

't 

Et»+,  Rt«+ 

It-Et/Rt 

♦♦Impossible  to  diagnose 


The  scoring  at  a  particular  point  is  based  on  the  correctness  of 
applying  the  appropriate  rule  using  preceding  conditions,  hence,  the  worng 
response  at  a  particular  point  does  not  effect  the  score  of  later 
variables.  This  scoring  enables  us  to  identify  the  exact  spot  where  an 
erroneous  application  of  a  rule  was  aade.  For  example,  the  following 
student  produced  the  response  pattern  shown  in  Table  5. 


Insert  Table  5  about  here 


The  example  given  in  Table  S  can  be  diagnosed  as  follows! 

1)  The  student  lacks  the  knowledge  that  Ea  ■  remains  fixed) 

2)  at  Ej  »  Ij  x  Rj,  substituting  "unchanged"  and  "decrease"  in  Rj  and 
I},  respectively,  Et  should  be  decreased.  The  same  action  is  repeated  at  E 

3)  at  ■  Et/Rt,  the  answer  should  be  undetermined,  but  the 
response  is  The  student's  error  is  due  to  a  lack  of  understanding  of 
the  algebraic  relationships  of  Ohm's  law. 

Discussion 

This  study  displayed  the  successful  application  of  the  component 
scoring  method  in  the  domain  of  electronics,  which  is  more  complicated  ther 
the  signed-number  subtraction  domain.  This  scoring  method  is  applicable 
and  useful  unless  the  number  of  components  is  unmanageably  large. 

The  immediate  return  of  component  scoring  is  being  able  to  identify 
where  the  error  was  made.  This  leads  directly  to  a  remediation  scheme  for 
a  particular  error.  The  evaluation  of  component  scores  is  difficult  to  do. 
One  way  to  do  this  evaluation  is  to  ask  how  objective  the  scoring  is  by 
examining  sources  of  variations  of  scoring  and  the  meaningfulness  of  these 


Table  5 


Coaparlion  between  coaponent  acorino  and  traditional  ‘right  or  wrong" 
acorinq  procedurea 


Problee 


R* 


E 

E 

E 

E 

I 

I 

I 

1 


1 

2 

3 

a 

1 

2 

3 

T 


Anewer 

♦ 

0 

♦ 

0 

♦ 


Coaponent  Traditional  Right 

Scoring _ Scoring _ An  ewer 

1  1  ♦ 

0  0  - 

1  1  ♦ 

0  0  - 

0  0  0 

1  1 


1  1 
1  1 
0  1 


.V  ,y 


-  -  </-  -  <  -  r-  if-  » 


•**  •’*  .*•  .*  *' 

•*.**V  “a*.'  V-V*V-  • 

Id  ■  ■  *  I  ,  -  .  ■  4  •  ,  ’  »  •  ,  ' 

-V-  «.'■  •A.'^i'VV 4  l* 


variations.  Using  some  information  similar  to  the  table,  one  can  answer 
these  two  questions.  In  this  study  there  was  only  one  general  sequential 
path,  and  the  variation  within  the  general  path  produced  identical  results, 
hence  variation  was  trivial. 

The  other  way  to  evaluate  the  scoring  is  to  compare  the  item  by  item 
correlation  matrix  of  component  scoring  and  regular  scoring.  If  the  scores 
reflect  the  knowledge  of  electricity  circuits,  then  the  correlations  of 
scores  should  be  non-negative.  It  was  found  that  there  were  eleven 
significantly  large  negative  correlations  for  the  regular  scoring,  while 
only  one  in  the  component  scoring. 

The  number  of  components  can  be  derived  by  first  making  a  theoretical 
study  of  materials  to  be  tested  and  further  modified  after  the  pilot  study, 
and  then  validating  the  components  by  comparing  the  real  data  and  the 
responses  simulated  by  the  erroneous  rules  of  various  components.  Benefits 
of  component  scoring  are  far  reaching.  For  example,  instead  of  a  very 
specific  conditional  description  needed  to  express  the  erroneous  rule  in 
one  component  (right  or  wrong  responses),  the  component  scoring  method  can 
express  most  of  the  erroneous  rules  by  the  combination  of  components  and 
component  specific  errors.  For  example  if  three  errors  are  identified  with 
each  of  four  components  then  3#*4»B1  combinations  of  possible  erroneous 
rules  exist;  and  each  of  81  erroneous  rules  can  be  specifically  defined  by 
only  12  rules.  Other  benefits  of  the  component  scoring  method  are  realized 
when  responses  are  mapped  on  to  the  rule  space  (Tatsuoka,  1984)  by  using 
right  or  wrong  response  patterns.  Some  rules  can  have  small  spatial 
distances  on  the  rule  space  but  the  response  patterns  are  distinctly 
unique.  For  such  a  case,  if  the  component  scoring  method  were  utilized, 
two  response  patterns  can  be  distinguished  more  precisely. 
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